Method and system for updating background model based on depth

ABSTRACT

A method and a system for updating a background model based on depth are disclosed. The method includes receiving, in response to the occurrence of a predetermined background updating condition, one or more depth images captured after a time when the predetermined background updating condition occurs; obtaining, based on an original background model, foreground images in the one or more captured depth images, which are newly added compared with a depth image at the time when the predetermined background updating condition occurs; for each of foreground pixels in each of the newly added foreground images, comparing a current depth value with a previous depth value before the time when the predetermined background updating condition occurs; and updating, when the current depth value is greater than the previous depth value, the original background model as the updated background model by using the foreground pixel in the newly added foreground image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to image processing, andspecifically, a method and a system for updating a background modelbased on depth.

2. Description of the Related Art

Background modeling (and subsequent foreground division) is veryimportant in detection and positioning of an object. However, there isinterference and disturbance in the process of background modeling. Asan important factor, a global or local illumination change, such as (butnot limited to) turn-on or turn-off of a lamp for controlling an officearea, temporarily covering of a lamp and opening or closing of acurtain, may lead to false foreground pixels. For example, in a casewhere an illumination condition becomes bright, a region wherebackground modeling is not originally performed may appear as a truebackground due to illumination increasing. However, the backgroundmodeling is not performed for such a region, thus this newly addedregion may be recognized as foreground pixels incorrectly. Accordingly,the background model is still not correct, and false foreground pixelsare recognized incorrectly.

FIG. 1 is a schematic drawing illustrating detected wrong foregroundpixels due to a sudden change of illumination situation in the priorart. As illustrated in FIG. 1, a large amount of false foreground pixelsoccur when illumination becomes bright. An RGB (Red, Green and Blue)image, a depth image and a foreground image on the left side areobtained under a relatively dark illumination condition, and an RGBimage, a depth image and a foreground image on the right side areobtained under a relatively bright illumination condition. Most of thepixels in the circle of the drawing on the left do not have a depthvalue because it is difficult to identify texture on an object surfacedue to relatively dark illumination; accordingly, depth values of thepixels that cannot be identified are set to 0. After the light becomesbright, a lot of pixels in the circle can be identified and obtain adepth value under a condition of texture enhancement. Thus, for thesepixels, a background model constructed under a dark illuminationcondition cannot match with depth values obtained under a relativelybright illumination condition; accordingly, a lot of false foregroundpixels illustrated in the drawing on the right side occur. These falseforeground pixels have a negative impact on further detection andtracking.

In fact, some methods of improving background modeling under a conditionof illumination changing have been proposed. However, most of thesemethods is applied to a RGB image and a case where the light changesslowly, and there is no method applied to a depth image and a case whereillumination changes rapidly (such as turn-on or turn-off). Therefore,the present invention has an object to solve or mitigate a problem ofbackground modeling of a depth image due to rapid change ofillumination.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, a method for updating abackground model based on depth includes: receiving, in response to theoccurrence of a predetermined background updating condition, one or moredepth images captured after a time when the predetermined backgroundupdating condition occurs; obtaining, based on an original backgroundmodel, foreground images in the one or more captured depth images, whichare newly added compared with a depth image at the time when thepredetermined background updating condition occurs; for each offoreground pixels in each of the newly added foreground images,comparing a current depth value with a previous depth value before thetime when the predetermined background updating condition occurs; andupdating, when the current depth value is greater than the previousdepth value, the original background model as the updated backgroundmodel by using the foreground pixel in the newly added foreground image.

According to another aspect of the present invention, a system forupdating a background model based on depth includes: a receptionapparatus configured to receive, in response to the occurrence of apredetermined background updating condition, one or more depth imagescaptured after a time when the predetermined background updatingcondition occurs; an obtainment apparatus configured to obtain, based onan original background model, foreground images in the one or morecaptured depth images, which are newly added compared with a depth imageat the time when the predetermined background updating condition occurs;a comparison apparatus configured to compare, for each of foregroundpixels in each of the newly added foreground images, a current depthvalue with a previous depth value before the time when the predeterminedbackground updating condition occurs; and an update apparatus configuredto update, when the current depth value is greater than the previousdepth value, the original background model as the updated backgroundmodel by using the foreground pixel in the newly added foreground image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic drawing illustrating detected wrong foregroundpixels due to a sudden change of illumination situation in the priorart;

FIG. 2 is a schematic drawing illustrating a hardware environment towhich a method of an embodiment of the present invention is applied;

FIG. 3 is a flowchart illustrating a method for updating a backgroundmodel based on depth according to an embodiment of the presentinvention;

FIG. 4 is a flowchart illustrating a method for updating a backgroundmodel based on depth according to another embodiment of the presentinvention;

FIG. 5 is a schematic drawing illustrating brightness increasing due tochange of an illumination condition;

FIG. 6 is a schematic drawing illustrating a relationship between trueforeground depth and true background depth;

FIG. 7 is a schematic drawing illustrating increasing of depth values ofpixels after the illumination condition changes; and

FIG. 8 is a block diagram illustrating a system for updating abackground model based on depth according to an embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following, embodiments of the present invention are described indetail with reference to the accompanying drawings, so as to facilitatethe understanding of the present invention. It should be understoodthat, the present invention is not limited to the embodiments, and thescope of the present invention may include various modifications,replacements or combinations. It should be noted that, the steps of themethod described here may be implemented by any functional block orfunctional design, and the functional block or functional design may beimplemented as a physical entity, a logical entity or a combinationthereof.

In the following, embodiments of the present invention are described indetail with reference to the accompanying drawings, so as to facilitatethe understanding of the present invention.

A background modeling method (namely, a foreground division method) isvery important for object recognition (type identification) and trackingrecovery serving as a final objective, since the background modeling canprovide 3D projecting points necessary for detection and tracking of anobject. There is disturbance in the modeling process.

Per-pixel background modeling may be divided into a static backgroundmodeling method and a dynamic background modeling method. In the staticmodeling method, a depth image is input, and an average value, i.e.,variation (variance) of a depth value at a predetermined pixel isexpressed by a single Gaussian model. In the dynamic modeling, a RGBimage is input, and change of a depth value at a pixel is expressed by aMulti-Gaussian model. Obviously, in the static background modeling, onlyone peak value occurs in this model because of limitations of the singleGaussian model, and statistical characteristics of the depth valuesduring a certain period of time are represented. Meanwhile, there are aplurality of Gaussian models in the dynamic background modeling, andlinear combination of these Gaussian models represents statisticalcharacteristics of the depth values during a certain period of time.

As the cause of the above problem, depth information has a uniquephysical distance that does not change in response to illuminationchange under an ideal situation; however, in actuality, relatively darkillumination may make the texture of a detection object become weak (forexample, the contrast of a pixel and local region thereof becomes weak).Accordingly, pixels where the left and right drawings cannot be matchedin a disparity search stage become invalid points, and their depthvalues may be set to 0. The background model constructed by using thesedepth images (using a method of taking statistics of average value andvariance pixel by pixel for many frames of depth images) cannot adapt tonewly obtained relatively accurate depth values of the invalid points,which matching between the left and the right can be performed whenillumination becomes bright. Thus, a large amount of false foregroundpoints are further produced, and it is difficult to meet the practicalapplication need.

Furthermore, after an original background model is constructed, a movingobject (such as a person) of true foreground pixels may exist in thescene. Thus, it is necessary to remove false foreground pixels due tosudden change of illumination, and to retain the true foreground pixels.

Actually, this is not a simple problem of illumination change.Specifically, illumination change is just a trigger factor of thisproblem. As illustrated in FIG. 1, true foreground pixels are caused bya motion of an object of interest (a person), and the pixels in thecircle of the lower-right drawing of FIG. 1 (false foreground pixels)are caused by non-matching between an old model and new data triggeredby an illumination condition. Thus, in order to detect an object (aperson) in the scene, it is necessary to remove false foreground pixelsand retain as many true foreground pixels as possible. Two extremesituations are considered. One is that all of the foreground pixels areremoved. In this case, all of the true foreground pixels are removed,although all of the false foreground pixels are removed. The other oneis foreground pixels are retained. In this case, all of the falseforeground pixels cannot be removed, although all of the true foregroundpixels are retained. Thus, this is a paradox.

In order to solve this technical problem, it is necessary to build anadaptive scheme for illumination, by which false foreground pixels canbe removed and true foreground pixels caused by object movement can beretained when illumination becomes bright.

Two points are considered to solve this problem.

(1) A predetermined condition is used for triggering an accumulationprocess to position newly occurring foreground pixels (for example, anillumination detection unit triggers update of the background under anideal condition only when it is detected that the illumination suddenlybecomes bright), and then a background model is always updated pixel bypixel by using current depth value. Two kinds of foreground pixels, thatis, true foreground pixels caused by object movement and falseforeground pixels caused by illumination change may be generated. Forthese two kinds of foreground pixels, it is necessary to provide aunified processing method to classify them formally.

(2) A physical principle is considered as follows. For a predeterminedpixel position, the depth value of true foreground is always smallerthan that of background. Accordingly, at a pixel level, a moving objectmay expose background pixels and generate relatively large depth values.Thus, the basis of target pixels for updating a background model isprovided.

The present invention may be made from following viewpoints based on theidea of the basic concept.

(1) A predetermined condition is detected to trigger an accumulationprocess of positioning of newly emerging foreground pixels.

(2) A depth comparison is performed for each of pixels to eliminateincorrect model updating, namely, only moving foreground pixels areupdated.

Accordingly, a background model constructed under a relatively darkcondition can be updated after the illumination condition becomesbetter. Thus, obviously incorrect foreground pixels do not appear aftersuch a period elapses, and the updated background is robust for rapidchange of illumination.

FIG. 2 is a schematic drawing illustrating hardware environment to whicha method of an embodiment of the present invention is applied.

As illustrated in FIG. 2, a stereo camera (such as a binocular camera)sends a captured stereo image to a decoder; the decoder decodes thestereo image and converts it into any required images, such as disparitymaps (namely, depth images), grayscale images or the like; a digitalsignal processor receives a target image (including but not beinglimited to a disparity map or a grayscale image) as an input image, andperforms interactive processing together with a storage apparatus tooutput an updated background model and other relevant data to a vehiclecontrol module; and the vehicle control module performs control of avehicle, such as foreground detection, pedestrian recognition, vehiclerecognition, road recognition, automatic driving or the like, based on adetected target and other relevant data. The digital signal processormay be connected to an input apparatus (the binocular camera or thedecoder), an output apparatus (the vehicle control module) and thestorage apparatus. The method provided by the present disclosure may beperformed in the digital signal processor.

FIG. 3 is a flowchart illustrating a method for updating a backgroundmodel based on depth according to an embodiment of the presentinvention.

As illustrated in FIG. 3, the method for updating a background modelbased on depth includes: step 301, receiving, in response to theoccurrence of a predetermined background updating condition, one or moredepth images captured after a time when the predetermined backgroundupdating condition occurs; step 302, obtaining, based on an originalbackground model, foreground images in the one or more captured depthimages, which are newly added compared with a depth image at the timewhen the predetermined background updating condition occurs; step 303,for each of foreground pixels in each of the newly added foregroundimages, comparing a current depth value with a previous depth valuebefore the time when the predetermined background updating conditionoccurs; and step 304, updating, when the current depth value is greaterthan the previous depth value, the original background model as theupdated background model by using the foreground pixel in the newlyadded foreground image.

In this way, the original background model can be updated as the updatedbackground using the foreground pixels whose the depth value is greaterthan an original depth value, by comparing the depth values of theforeground pixels in the newly added foreground image with the originaldepth value, after the predetermined background updating conditionOccurs. Thus, the background model can be adaptively updated.

In an embodiment, whether the predetermined background updatingcondition occurs may be determined based on at least one of theconditions: whether an increasing amount or an increasing rate ofenvironment brightness is greater than a predetermined threshold,whether a change amount or a change rate of environment brightness isgreater than a predetermined threshold, and whether a predeterminedperiod has elapsed. For example, when the illumination conditionsuddenly becomes bright, namely, the increasing amount or the increasingrate of brightness is greater than the predetermined threshold, a truebackground appears because the illumination condition becomes better.However, the newly added region may be recognized as foreground pixelsincorrectly since the background model is not constructed for thisregion; accordingly, the background model is still incorrect and falseforeground pixels are recognized incorrectly. Thus, the backgroundupdating method according to the embodiment of the present invention maybe performed at this time. It should be noted that, as a startingcondition of triggering of the background updating, the condition thatthe illumination condition becomes better is just an example; othertriggering conditions, such as regular updating, periodic updating orobtainment of an illumination control signal may also be applied to theupdating of the background model, but the present invention is notlimited to the above triggering condition.

Additionally, after the background model is updated one time, theupdating may be not performed when the illumination condition suddenlybecomes bright again, since the previous updating has already reachedappropriate effect. That is to say, the number of times of updating thebackground model may be set to 1. It should be noted that, the number oftimes of updating the background model may also be set to another value,or the background may be updated at each time when the illuminationcondition becomes bright.

In an embodiment, depth values of unmodeling pixels in the originalbackground model may be set to 0. In a case where illumination conditionis dark, it is difficult to identify the texture of a surface of anobject because of weak illumination (namely, left and right imagescaptured by a stereo camera are not matched); accordingly, the depthvalues of these pixels that cannot be identified are set to 0, namely,these pixels do not have depth value. It should be noted that, the depthvalues of these pixels that cannot be identified may also be set toother values to represent that a background model is not constructed forthese pixels.

In an embodiment, obtaining, based on the original background model, theforeground images in the one or more captured depth images, which arenewly added compared with the depth image at the time when thepredetermined background updating condition occurs (step 302) mayinclude: obtaining, based on the original background model, a firstforeground image in the depth image captured at the time when thepredetermined background updating condition occurs; obtaining, based onthe original background model, second foreground images in the one ormore depth images captured after a time when the predeterminedbackground updating condition occurs; performing subtraction betweeneach of the second foreground images and the first foreground image;setting newly added foreground images in the one or more captured depthimages obtained by the subtraction as the newly added foreground images;and omitting a vanishing foreground image and a cancelled foregroundimage in the depth images obtained by the subtraction.

Foreground pixels that change compared with an original depth image canbe obtained by performing subtraction between the one or more captureddepth images and the depth image captured before the time when thepredetermined background updating condition occurs. If the obtainedpixel is positive, the obtained pixel may be regarded as a newly addedforeground pixel; if the obtained pixel is negative, the obtained pixelmay be regarded as a vanishing foreground pixel; and if the obtainedpixel is cancelled, namely the obtained pixel is 0, the obtained pixelmay be regarded as a position without change. Thus, the vanishingforeground pixels and the cancelled foreground pixels may not beconsidered, because all of these foreground pixels are not thebackground newly added due to the change of the illumination condition.In this way, the comparison of the depth values is performed only forthe newly added foreground pixels that are the true background, thuscomputational cost and time can be reduced and operation speed can beaccelerated.

In an embodiment, wherein updating, when the current depth value isgreater than the previous depth value, the original background model asthe updated background model by using the foreground pixel in the newlyadded foreground image includes: updating the original background modelas the updated background model by using a position and a depth value ofthe foreground pixel in the newly added foreground image. Generally, anew background model may be obtained by updating the position and thedepth value of a pixel, however a grayscale value and a RGB value mayalso be used for updating to identify the background better.

In this way, the background model can be adaptively updated when thepredetermined background updating condition occurs, thus a more accuratebackground model can be constantly obtained.

In the following, a detailed example of background model updating may bedescribed with reference to the accompanying drawings. It should benoted that, FIG. 4 and subsequent drawings are just a detailed example,and the embodiment of the present invention is not limited to thefollowing detailed steps, values, conditions, data, order, etc. Otherembodiments unmentioned in the present specification may be made bypersons skilled in the art by reading the present specification andutilizing the idea of the present invention.

FIG. 4 is a flowchart illustrating a method for updating a backgroundmodel based on depth according to another embodiment of the presentinvention.

The method illustrated in FIG. 4 includes: S401, determining anillumination condition; S402, receiving depth images from N-th frame to(N+m)-th frame (N and m are positive integers) during an illuminationincreasing period; S403, obtaining a series of foreground images indepth images from N-th frame to (N+m)-th frame, based on an originalbackground model; S404, subtracting the foreground image of N-th framefrom each of the foreground images from (N+1)-th frame to (N+m)-th frameto position newly appearing foreground pixels in each of the m frames,respectively, that including true foreground image and false foregroundimage due to the change of the illumination; S405, determining whether acurrent depth value of the newly added foreground pixel is greater thanan original depth value for each of the pixels, and if the current depthvalue is greater than the original depth value, the process proceeds tostep S406, and if the current depth value is not greater than theoriginal depth value, the process returns to the next pixel to bedetermined; S406, using the current depth value as the updatedbackground model to remove incorrect models and update the model only bytrue foreground pixels due to movement.

In step S401, in determining the illumination condition, if rapidenhancement of illumination is detected as illustrated in the leftdrawing of FIG. 5 (FIG. 5 is a schematic drawing illustrating brightnessincreasing of change of an illumination condition), this may be regardedas the predetermined background updating condition to start the processof the background updating.

For example, an RGB image may be converted into a YUV (or YCbCr,Luminance-Chrominance) space, a Y channel representing a brightnesscomponent. Thus, rapid enhancement of illumination can be determined bydetecting a sharp rise of the Y brightness component (the left drawingof FIG. 5), and the rapid enhancement of illumination can be regarded asa trigger condition. During an illumination enhancement period, a modelof each of the pixels is updated by the current depth value. The rightdrawing of FIG. 5 illustrates a curve of illumination change in a room,in which three operations, turn-on, turn-off and turn-on again of a lampare included.

It should be noted that, after the background model is updated one time,the updating may be not performed when the illumination conditionsuddenly becomes bright again (the updating is not performed when thelamp is turned on again after it is turned off as illustrated in theright drawing of FIG. 5, namely, the light is enhanced for the secondtime); this is because the previous updating has already reachedappropriate effect. That is to say, the number of times of updating thebackground model may be set to 1. It should be noted that, the number oftimes of updating the background model may also be set to another value,or the background may be updated at each time when the illuminationcondition becomes bright.

Next, when a detection unit detects a rapid enhancement of illumination(as illustrated in the left drawing of FIG. 5) at the N-th frame, thedetection unit triggers the following accumulation to update thebackground model. For example, when an event of illumination enhancementis detected at the N-th frame, the N-th frame may be regarded asreference. Namely, the next m frames are received (step S402). Next, forthe received m frames (from the (N+1)-th frame to the (N+m)-th frame),foreground images based on an original background model are calculated(step S403) and the foreground image based on the original backgroundmodel at the N-th frame reference is subtracted from the foregroundimages of the m frames, so that newly added foreground pixels at each ofthe m frames are obtained (namely, the foreground pixels whose thesubtraction result is positive are obtained, and the foreground pixelswhose the subtraction result is 0 or negative are ignored) (step S404).In this way, a difference between a foreground mask under a relativelybright illumination condition and a foreground mask under a relativelydark illumination condition is saved as primary information of an updateposition (or region), so that the depth value that appears whenillumination becomes bright is fully utilized. In this way, “positiveparts” (namely, the foreground pixels whose the subtraction result ispositive) can be obtained; that is to say, the difference between theforeground mask under the relatively bright illumination condition andthe foreground mask under the relatively dark illumination condition issaved as the primary information of the update position (or region),even though the update position (or region) includes both trueforeground pixels due to object movement and false foreground pixels dueto illumination change. According to the embodiment, this information issaved, namely, difference of frames are accumulated on a special image(called a “positive parts image”). The reason why it is called “positiveparts” is because a pixel corresponding to 1 in the foreground maskunder a relatively bright illumination condition and a pixelcorresponding to 0 in the foreground mask under a relatively darkillumination condition are recorded as positive in a final accumulationimage (1−0=1), and non-positive values are obtained in other cases (forexample, 1−1=0, 0−0=0 and 0−1=−1) and ignored in the final accumulationimage. In this process, when the accumulation is performed for each ofthe pixels, depth values in the current “mask image” (not the wholeimage) of “positive parts” under relatively bright illumination are usedfor updating the original background model, so that a smooth updatingcan be ensured. In this way, more reliable depth data under a brightillumination condition can be fully utilized.

For a given pixel position, true foreground depth is generally smallerthan true background depth. For example, as illustrated in FIG. 6 (FIG.6 is a schematic drawing illustrating a relationship between trueforeground depth and true background depth), along the same projectionline, the background point Q′ is always blocked by the foreground pointQ, thus an image point (q) of Q rather than Q′ is generated on the imageplane. It should be noted that, if Q moves to a position behind Q′ andis still located on the projection line, Q will be blocked by Q′ and isno longer visible. At this time, an image of Q′ appears on the imageplane. This is the principle for providing candidate pixels to updatethe background model; at the pixel level, movement of an object mayexpose true background pixels, thus a depth value greater than anoriginal depth value may be generated. As illustrated by the arrow andthe indication word “move”, the foreground object Q moves along thedirection shown by the arrow, and the background Q′ appears. At thistime, if the depth value Z′ at this pixel position and the originaldepth value Z are compared (step S405), it may be found that the depthvalue Z′ at this pixel position is greater than the original depth valueZ. Accordingly, it may be determined that this pixel at position Q′ is abackground pixel, and the background model may be updated by this pixel,so that a more accurate background model is obtained (step S406).

FIG. 7 is a schematic drawing illustrating increasing of depth values ofpixels after the illumination condition changes. Erroneous modelupdating is removed by performing comparison for each pixel, and thebackground model is updated by true moving foreground pixels. In acamera coordinate system, if a position of an object changes to a newposition after N frames, the depth values of the pixels at the originalpositions of the object will increase. This is because the truebackground pixels at these positions are not blocked by the originalforeground but are exposed. It should be noted that, it is a process ofgradual accumulation of motion shift to make this process perform moresmoothly. As an example, the simplest policy is to use an obtainedmaximum depth value.

Background(current)=max(depth{N,N+1, . . . , Current})   (1)

Background(current) is a background model expressed by depth values,depth(x) represents a depth value of x frames (x∈[N,N+m]) and isobtained by calculating a maximum depth value from N-th frame to acurrent frame (namely, a frame of m frames). It should be noted that,besides the above formula, other policies (for example, for a pixel, anupdating time when the depth values are compared and the backgroundmodel is updated) may be used. That is to say, in the embodiment, in them frames after the time when the background updating condition occurs,only the maximum depth value is used as the depth value for updating thebackground (because the background usually is the deepest). However, itis just an example. Actually, only the depth value of the foregroundpixels of the m-th frame (namely, the last frame) may be used, or onlyan average value of the depth values of the foreground pixels in mframes may be used as the depth values for updating the background. Ofcourse, various designs may be performed based on accuracy andreliability of updating of the background model to be realized.

Moreover, a moving direction and speed of a foreground object in a spacewhen the predetermined background updating condition occurs may beconsidered. Specifically, if the moving direction and the speed aredifferent from a direction and a speed for exposing the background inthe usual case, it may be not considered that the background model isupdated by using the depth values of the foreground object; and if themoving direction and the speed are the same as the direction and thespeed for exposing the background in the usual case, it may beconsidered that the background model is updated by using the depthvalues of the foreground object. In this way, some situations in whichthe background is not exposed can be removed, and erroneous backgroundmodel updating can be removed.

FIG. 8 is a block diagram illustrating a system for updating abackground model based on depth according to an embodiment of thepresent invention.

The system for updating a background model based on depth 800illustrated in FIG. 8 includes: a reception apparatus 801 configured toreceive, in response to the occurrence of a predetermined backgroundupdating condition, one or more depth images captured after a time whenthe predetermined background updating condition occurs; an obtainmentapparatus 802 configured to obtain, based on an original backgroundmodel, foreground images in the one or more captured depth images, whichare newly added compared with a depth image at the time when thepredetermined background updating condition occurs; a comparisonapparatus 803 configured to compare, for each of foreground pixels ineach of the newly added foreground images, a current depth value with aprevious depth value before the time when the predetermined backgroundupdating condition occurs; and an update apparatus 804 configured toupdate, when the current depth value is greater than the previous depthvalue, the original background model as the updated background model byusing the foreground pixel in the newly added foreground image.

In this way, the original background model can be updated as the updatedbackground using the foreground pixels whose the depth value is greaterthan an original depth value, by comparing the depth values of theforeground pixels in the newly added foreground image with the originaldepth value, after the predetermined background updating conditionoccurs. Thus, the background mode can be adaptively updated.

In an embodiment, whether the predetermined background updatingcondition occurs may be determined based on at least one of theconditions: whether an increasing amount or an increasing rate ofenvironment brightness is greater than a predetermined threshold,whether a change amount or a change rate of environment brightness isgreater than a predetermined threshold, and whether a predeterminedperiod has elapsed. For example, when the illumination conditionsuddenly becomes bright, namely, the increasing amount or the increasingrate of brightness is greater than the predetermined threshold, a truebackground appears because the illumination condition becomes better.However, the newly added region may be recognized as foreground pixelsincorrectly since the background model is not constructed for thisregion; accordingly, the background model is still incorrect and falseforeground pixels are recognized incorrectly. Thus, the backgroundupdating method according to the embodiment of the present invention maybe performed at this time. It should be noted that, as a startingcondition of triggering the background updating, the condition that theillumination condition becomes better is just an example; othertriggering conditions, such as regular updating, periodic updating orobtainment of an illumination control signal may also be applied to theupdating of the background model, but the present invention is notlimited to the above triggering condition.

Additionally, after the background model is updated one time, theupdating may be not performed when the illumination condition suddenlybecomes bright again, since the previous updating has already reachedappropriate effect. That is to say, the number of times of updating thebackground model may be set to 1. It should be noted that, the number oftimes of updating the background model may also be set to another value,or the background may be updated at each time when the illuminationcondition becomes bright.

In an embodiment, depth values of unmodeled pixels in the originalbackground model may be set to 0. In a case where illumination conditionis dark, it is difficult to identify the texture of a surface of anobject because of weak illumination (namely, left and right imagescaptured by a stereo camera are not matched); accordingly, the depthvalues of these pixels that cannot be identified are set to 0, namely,these pixels do not have depth value. It should be noted that, the depthvalues of these pixels that cannot be identified may also be set toother values to represent that a background model is not constructed forthese pixels.

In an embodiment, the obtainment apparatus 802 may obtain, based on theoriginal background model, a first foreground image in the depth imagecaptured at the time when the predetermined background updatingcondition occurs; obtain, based on the original background model, secondforeground images in the one or more depth images captured after a timewhen the predetermined background updating condition occurs; performsubtraction between each of the second foreground images and the firstforeground image; set newly added foreground images in the one or morecaptured depth images obtained by the subtraction as the newly addedforeground images; and omit a vanishing foreground image and a cancelledforeground image in the depth images obtained by the subtraction.

Foreground pixels that change compared with an original depth image canbe obtained by performing subtraction between the one or more captureddepth images and the depth image captured before the time when thepredetermined background updating condition occurs. If the obtainedpixel is positive, the obtained pixel may be regarded as a newly addedforeground pixel; if the obtained pixel is negative, the obtained pixelmay be regarded as a vanishing foreground pixel; and if the obtainedpixel is cancelled, namely the obtained pixel is 0, the obtained pixelmay be regarded as a position without change. Thus, the vanishingforeground pixels and the cancelled foreground pixels may not beconsidered, because all of these foreground pixels are not thebackground newly added due to the change of the illumination condition.In this way, the comparison of the depth values is performed only forthe newly added foreground pixels that are the true background, thuscomputational cost and time can be reduced and operation speed can beaccelerated.

In an embodiment, wherein updating, when the current depth value isgreater than the previous depth value, the original background model asthe updated background model by using the foreground pixel in the newlyadded foreground image includes: updating the original background modelas the updated background model by using a position and a depth value ofthe foreground pixel in the newly added foreground image. Generally, anew background model may be obtained by updating the position and thedepth value of a pixel, however a grayscale value and a RGB value mayalso be used for updating to identify the background better.

In this way, the background model can be adaptively updated when thepredetermined background updating condition occurs, thus a more accuratebackground model can be constantly obtained.

According to the embodiments of the present invention, the updating ofthe background model can be adaptively started when the determinedbackground updating condition occurs, thus a more accurate backgroundmodel can be obtained. In order to measure the accuracy, three technicalindicators are defined.

Technical Indicator 1: SNR=signal/noise, which represents thesignificance of the foreground signal with respect to the noise.

Technical indicator 2: Hit Rate=signal/true signal, which represents thestorage capability for true foreground information.

Technical indicator 3: operating time, which represents the real-timeperformance of the system.

As proved by experiments, according to the embodiments of the presentinvention, SNR can be increased, an acceptable Hit Rate can bemaintained and little time is consumed. For example, as shown in data ofan experiment, generally, according to the embodiments of the presentinvention, SNR can be increased five times and HitRate can be maintainedat 80% of the original HitRate.

It should be noted that, the above specific embodiments are justexamples and the present invention is not limited to those embodiments.The embodiments may be combined or some steps or apparatuses may bemerged according to the embodiments by persons skilled in the artaccording to the concept of the present invention to realize the effectof the present invention, and the combined embodiments and the mergedembodiments are included within the present invention. The descriptionof the combination and merging is omitted here.

It should be noted that, the advantage or effect of the presentinvention is described above. The above descriptions of the embodimentsare just examples, and various modifications, replacements orcombinations may be made without departing from the scope of the presentinvention by persons skilled in the art.

The block diagrams of the units, apparatuses, devices and system arejust examples, the connection, placement and configuration illustratedin the block diagrams related to the present invention are not limitedto these examples, and the units, apparatuses, devices and system may beconnected, placed or configured in any way. The terms “comprise”,“include” and “have” are open-form terms, which mean and may be changedinto “include and is not limited to”. The terms “or” and “and” mean andmay be changed into “and/or”, unless the context is clearly not. Theterm “such as” means and may be changed to “such as, but not limitedto”.

The flowchart and the method according to the present invention are justexamples, and not limited to the steps in the embodiments. The steps ofthe embodiments may be performed in any order. The terms “next”,“subsequently” and “then” are just for describing the present invention,and the present invention is not limited to these terms. Furthermore,the articles “a”, “an” and “the” should not be limited to the singularelement.

The steps or apparatuses of the present invention are described above.The above descriptions of the embodiments are just examples, and variousmodifications, replacements or combinations may be made withoutdeparting from the scope of the present invention by persons skilled inthe art.

The steps of the above method may be performed by any appropriate meansthat can perform the corresponding functions. The means may include anycomponents and/or modules of hardware and/or software, and include butnot be limited to a circuit, a dedicated application-specific integratedcircuit (ASIC) or a processor.

The present invention may use a general-purpose processor, a digitalsignal processor (DSP), an ASIC, a field programmable gate array (FPGA)or other programmable logic device (PLD), a discrete gate or transistorlogic, discrete hardware components or any other combination forexecuting the functions to realize the logic blocks, modules andcircuits of the embodiments. The general-purpose processor is amicro-processor, and alternatively, the processor may be any processors,controllers, micro-controllers or state machines that can be obtainedcommercially. The processor may also be the combination of the computerequipment, such as the combination of a DSP and a micro-processor, thecombination of plural micro-processors, or the combination of a DSP andplural micro-processors.

The steps of the method according to the present invention may beincorporated in the hardware, software modules executed by a processoror the combination of these two directly. The software modules may bestored in a recording medium with any shape. The examples of therecording medium includes a random access memory (RAM), a read-onlymemory (ROM), a flash memory, an EPROM memory, an EEPROM memory, aregister, a hard disk drive, a removable disk, a CD-ROM, etc. Therecording medium may be linked to a processor so that the processorreads information from the recording medium or writes information intothe recording medium. Alternatively, the recording medium and theprocessor may also be a whole apparatus. The software module may be asingle command or many commands, and may be distributed in several codesegments, different programs or plural recording media.

Steps of the above method may be performed in time order, however theperforming sequence is not limited to the time order. Any steps may beperformed in parallel or independently.

The functions may be realized by hardware, software, firmware or anycombination thereof. When the function is implemented by software, thefunction may be stored in a computer-readable medium as one or morecommands. The recording medium may be any real medium that can beaccessed by a computer. Such a computer-readable medium includes a RAM,a ROM, an EEPROM, a CD-ROM or other laser discs, a magnetic disk orother magnetic memory, or any other real media that carry or storecommands, data or program codes and are accessed by the computer. Suchdisk and disc include a CD, a laser disc, an optical disc, a DVD disc, afloppy disk and a blue-ray disc, and the disk usually reproduces dataand the disc reproduces data by a laser.

Thus, the operations may be performed by a computer program product. Forexample, such computer program product may be a tangible medium wherecomputer-readable commands are stored (or coded), and the commands maybe executed by one or more processors to perform the operation. Thecomputer program product may include packaging material.

The software or commands may also be transmitted by a transmissionmedium. For example, a coaxial cable, an optical cable, a twisted cable,a digital subscriber line (DSL), or a transmission medium of thewireless technology of infrared, wireless or microwave may be used totransmit the software from a website, a server or another remote source.

Additionally, the modules and/or other appropriate means of the methodor technology may be obtained from a user terminal and/or base station,or by other methods. For example, such equipment may be connected to aserver so as to perform the transmission of the means of the abovemethod. Alternatively, the methods may be provided via a storage unit(for example, a physical storage medium such as a RAM, a ROM, a CD or afloppy disc), so that the user terminal and/or the base station canobtain the methods when it is connected to the equipment. Furthermore,any other appropriate technology may be provided to the equipment by themethod.

The present specification and the appended claims includes otherexamples and implementations. For example, the above functions may beimplemented by a processor, hardware, software, firmware, hard-wire orany combination thereof. The features for implementing the functions maybe located at any physical position where which is distributed to eachposition physically. Furthermore, the term “or” before the term “atleast one” means a separate enumerating, and for example, “at least oneof A, B or C” means (1) A, B or C, (2) AB, AC or BC, or (3) ABC (namely,A and B and C). Additionally, the term “example” does not mean apreferable example or an example superior to other examples.

Various modifications, replacements or combinations may be made withoutdeparting from the scope of the present invention by persons skilled inthe art. Furthermore, the scope of the present specification and theclaims are not limited to the above processing, machine, manufacture,composition of events, means, method and operation. The processing,machine, manufacture, composition of events, means, method and operationwith a similar function or a similar result may also be applied to thepresent invention. Therefore, the scope of the appended claims includesuch processing, machine, manufacture, composition of events, means,method and operation.

The purposes of the present invention is described above. The abovedescriptions of the embodiments are just examples, and variousmodifications, replacements or combinations may be made withoutdeparting from the scope of the present invention by persons skilled inthe art.

The basic principle of the present invention is described above withreference to the embodiments, however the present invention is notlimited to the principle.

The present application is based on and claims the benefit of priorityof Chinese Priority Application No. 201410283056.2 filed on Jun. 23,2014, the entire contents of which are hereby incorporated by reference.

What is claimed is:
 1. A method for updating a background model based ondepth, the method comprising: receiving, in response to the occurrenceof a predetermined background updating condition, one or more depthimages captured after a time when the predetermined background updatingcondition occurs; obtaining, based on an original background model,foreground images in the one or more captured depth images, which arenewly added compared with a depth image at the time when thepredetermined background updating condition occurs; for each offoreground pixels in each of the newly added foreground images,comparing a current depth value with a previous depth value before thetime when the predetermined background updating condition occurs; andupdating, when the current depth value is greater than the previousdepth value, the original background model as the updated backgroundmodel by using the foreground pixel in the newly added foreground image.2. The method for updating a background model according to claim 1,wherein whether the predetermined background updating condition occursis determined based on at least one of whether an increasing amount oran increasing rate of environment brightness is greater than apredetermined threshold, whether a change amount or a change rate ofenvironment brightness is greater than a predetermined threshold, andwhether a predetermined period has elapsed.
 3. The method for updating abackground model according to claim 1, wherein depth values ofunmodeling pixels in the original background model are set to
 0. 4. Themethod for updating a background model according to claim 1, whereinobtaining, based on the original background model, the foreground imagesin the one or more captured depth images, which are newly added comparedwith the depth image at the time when the predetermined backgroundupdating condition occurs includes obtaining, based on the originalbackground model, a first foreground image in the depth image capturedat the time when the predetermined background updating condition occurs;obtaining, based on the original background model, second foregroundimages in the one or more depth images captured after a time when thepredetermined background updating condition occurs; performingsubtraction between each of the second foreground images and the firstforeground image; setting newly added foreground images in the one ormore captured depth images obtained by the subtraction as the newlyadded foreground images; and omitting a vanishing foreground image and acancelled foreground image in the depth images obtained by thesubtraction.
 5. The method for updating a background model according toclaim 1, wherein updating, when the current depth value is greater thanthe previous depth value, the original background model as the updatedbackground model by using the foreground pixel in the newly addedforeground image includes updating the original background model as theupdated background model by using a position and a depth value of theforeground pixel in the newly added foreground image.
 6. The method forupdating a background model according to claim 1, wherein the number oftimes of updating the background model is set to a positive integer. 7.The method for updating a background model according to claim 2, whereinthe environment brightness is obtained from a brightness value obtainedby converting a currently captured depth image into aluminance-chrominance space.
 8. A system for updating a background modelbased on depth, the system comprising: a reception apparatus configuredto receive, in response to the occurrence of a predetermined backgroundupdating condition, one or more depth images captured after a time whenthe predetermined background updating condition occurs; an obtainmentapparatus configured to obtain, based on an original background model,foreground images in the one or more captured depth images, which arenewly added compared with a depth image at the time when thepredetermined background updating condition occurs; a comparisonapparatus configured to compare, for each of foreground pixels in eachof the newly added foreground images, a current depth value with aprevious depth value before the time when the predetermined backgroundupdating condition occurs; and an update apparatus configured to update,when the current depth value is greater than the previous depth value,the original background model as the updated background model by usingthe foreground pixel in the newly added foreground image.
 9. The systemfor updating a background model according to claim 8, wherein whetherthe predetermined background updating condition occurs is determinedbased on at least one of whether an increasing amount or an increasingrate of environment brightness is greater than a predeterminedthreshold, whether a change amount or a change rate of environmentbrightness is greater than a predetermined threshold, and whether apredetermined period has elapsed.
 10. The system for updating abackground model according to claim 8, wherein depth values ofunmodeling pixels in the original background model are set to
 0. 11. Thesystem for updating a background model according to claim 8, wherein theobtainment apparatus obtains, based on the original background model, afirst foreground image in the depth image captured at the time when thepredetermined background updating condition occurs; obtains, based onthe original background model, second foreground images in the one ormore depth images captured after a time when the predeterminedbackground updating condition occurs; performs subtraction between eachof the second foreground images and the first foreground image; setsnewly added foreground images in the one or more captured depth imagesobtained by the subtraction as the newly added foreground images; andomits a vanishing foreground image and a cancelled foreground image inthe depth images obtained by the subtraction.
 12. The system forupdating a background model according to claim 8, wherein the updateapparatus updates the original background model as the updatedbackground model by using a position and a depth value of the foregroundpixel in the newly added foreground image.
 13. The system for updating abackground model according to claim 8, wherein the number of times ofupdating the background model is set to a positive integer.
 14. Thesystem for updating a background model according to claim 9, wherein theenvironment brightness is obtained from a brightness value obtained byconverting a currently captured depth image into a luminance-chrominancespace.